{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "multi-class_classification_of_handwritten_digits.ipynb",
      "version": "0.3.2",
      "views": {},
      "default_view": {},
      "provenance": [],
      "collapsed_sections": [
        "266KQvZoMxMv",
        "6sfw3LH0Oycm",
        "copyright-notice"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "copyright-notice",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Copyright 2017 Google LLC."
      ]
    },
    {
      "metadata": {
        "colab": {
          "autoexec": {
            "wait_interval": 0,
            "startup": false
          }
        },
        "id": "copyright-notice2",
        "colab_type": "code",
        "cellView": "both"
      },
      "cell_type": "code",
      "outputs": [],
      "execution_count": 0,
      "source": [
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "metadata": {
        "id": "mPa95uXvcpcn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " # Classer des chiffres \u00e9crits \u00e0 la main avec les r\u00e9seaux de neurones"
      ]
    },
    {
      "metadata": {
        "id": "Fdpn8b90u8Tp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ![img](https://www.tensorflow.org/versions/r0.11/images/MNIST.png)"
      ]
    },
    {
      "metadata": {
        "id": "c7HLCm66Cs2p",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " **Objectifs d'apprentissage\u00a0:**\n",
        "  * Entra\u00eener un mod\u00e8le lin\u00e9aire et un r\u00e9seau de neurones \u00e0 classer des chiffres \u00e9crits \u00e0 la main \u00e0 partir de l'ensemble de donn\u00e9es [MNIST](http://yann.lecun.com/exdb/mnist/) classique\n",
        "  * Comparer les performances des mod\u00e8les de classification lin\u00e9aire et avec r\u00e9seau de neurones\n",
        "  * Visualiser les pond\u00e9rations d'une couche cach\u00e9e de r\u00e9seau de neurones"
      ]
    },
    {
      "metadata": {
        "id": "HSEh-gNdu8T0",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " L'objectif est d'associer \u00e0 chaque image d'entr\u00e9e le chiffre num\u00e9rique correct. Vous allez cr\u00e9er un r\u00e9seau de neurones avec quelques couches cach\u00e9es et, au sommet, une couche Softmax afin de s\u00e9lectionner la classe qui l'emporte."
      ]
    },
    {
      "metadata": {
        "id": "2NMdE1b-7UIH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ## Configuration\n",
        "\n",
        "Commencez par t\u00e9l\u00e9charger l'ensemble de donn\u00e9es, importez TensorFlow et d'autres utilitaires, puis chargez les donn\u00e9es dans un `DataFrame` *Pandas*. Notez que ces donn\u00e9es constituent un \u00e9chantillon des donn\u00e9es d'apprentissage MNIST d'origine\u00a0; 20\u00a0000\u00a0lignes ont \u00e9t\u00e9 prises au hasard."
      ]
    },
    {
      "metadata": {
        "id": "4LJ4SD8BWHeh",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "test": {
            "output": "ignore",
            "timeout": 600
          }
        },
        "cellView": "both"
      },
      "source": [
        "from __future__ import print_function\n",
        "\n",
        "import glob\n",
        "import math\n",
        "import os\n",
        "\n",
        "from IPython import display\n",
        "from matplotlib import cm\n",
        "from matplotlib import gridspec\n",
        "from matplotlib import pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import seaborn as sns\n",
        "from sklearn import metrics\n",
        "import tensorflow as tf\n",
        "from tensorflow.python.data import Dataset\n",
        "\n",
        "tf.logging.set_verbosity(tf.logging.ERROR)\n",
        "pd.options.display.max_rows = 10\n",
        "pd.options.display.float_format = '{:.1f}'.format\n",
        "\n",
        "mnist_dataframe = pd.read_csv(\n",
        "  \"https://download.mlcc.google.com/mledu-datasets/mnist_train_small.csv\",\n",
        "  sep=\",\",\n",
        "  header=None)\n",
        "\n",
        "# Use just the first 10,000 records for training/validation.\n",
        "mnist_dataframe = mnist_dataframe.head(10000)\n",
        "\n",
        "mnist_dataframe = mnist_dataframe.reindex(np.random.permutation(mnist_dataframe.index))\n",
        "mnist_dataframe.head()"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "kg0-25p2mOi0",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " La premi\u00e8re colonne contient l'\u00e9tiquette de classe. Les autres contiennent les valeurs de caract\u00e9ristique\u00a0; une par pixel pour les `28\u00d728=784`\u00a0valeurs de pixel. La plupart de ces 784\u00a0valeurs de pixel sont \u00e9gales \u00e0 z\u00e9ro\u00a0; vous pouvez prendre une minute pour v\u00e9rifier qu'elles ne sont pas *toutes* nulles."
      ]
    },
    {
      "metadata": {
        "id": "PQ7vuOwRCsZ1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ![img](https://www.tensorflow.org/versions/r0.11/images/MNIST-Matrix.png)"
      ]
    },
    {
      "metadata": {
        "id": "dghlqJPIu8UM",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " Ces exemples sont, en fait, des images \u00e0 fort contraste et \u00e0 relativement faible r\u00e9solution de nombres \u00e9crits \u00e0 la main. Chacun des dix chiffres `0-9` est repr\u00e9sent\u00e9 avec une \u00e9tiquette de classe unique. Il s'agit donc d'un probl\u00e8me de classification \u00e0 classes multiples avec 10\u00a0classes.\n",
        "\n",
        "Nous allons, \u00e0 pr\u00e9sent, analyser les \u00e9tiquettes et les caract\u00e9ristiques, et observer quelques exemples. Notez l'utilisation de `loc` qui permet d'extraire des colonnes en fonction de l'emplacement d'origine, \u00e9tant donn\u00e9 que cet ensemble de donn\u00e9es est d\u00e9pourvu de ligne d'en-t\u00eate."
      ]
    },
    {
      "metadata": {
        "id": "JfFWWvMWDFrR",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "test": {
            "output": "ignore",
            "timeout": 600
          }
        }
      },
      "source": [
        "def parse_labels_and_features(dataset):\n",
        "  \"\"\"Extracts labels and features.\n",
        "  \n",
        "  This is a good place to scale or transform the features if needed.\n",
        "  \n",
        "  Args:\n",
        "    dataset: A Pandas `Dataframe`, containing the label on the first column and\n",
        "      monochrome pixel values on the remaining columns, in row major order.\n",
        "  Returns:\n",
        "    A `tuple` `(labels, features)`:\n",
        "      labels: A Pandas `Series`.\n",
        "      features: A Pandas `DataFrame`.\n",
        "  \"\"\"\n",
        "  labels = dataset[0]\n",
        "\n",
        "  # DataFrame.loc index ranges are inclusive at both ends.\n",
        "  features = dataset.loc[:,1:784]\n",
        "  # Scale the data to [0, 1] by dividing out the max value, 255.\n",
        "  features = features / 255\n",
        "\n",
        "  return labels, features"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "mFY_-7vZu8UU",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "training_targets, training_examples = parse_labels_and_features(mnist_dataframe[:7500])\n",
        "training_examples.describe()"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4-Vgg-1zu8Ud",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "validation_targets, validation_examples = parse_labels_and_features(mnist_dataframe[7500:10000])\n",
        "validation_examples.describe()"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "wrnAI1v6u8Uh",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " Affichez un exemple al\u00e9atoire et l'\u00e9tiquette correspondante."
      ]
    },
    {
      "metadata": {
        "id": "s-euVJVtu8Ui",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "rand_example = np.random.choice(training_examples.index)\n",
        "_, ax = plt.subplots()\n",
        "ax.matshow(training_examples.loc[rand_example].values.reshape(28, 28))\n",
        "ax.set_title(\"Label: %i\" % training_targets.loc[rand_example])\n",
        "ax.grid(False)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ScmYX7xdZMXE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ## T\u00e2che\u00a01\u00a0: Construire un mod\u00e8le lin\u00e9aire pour MNIST\n",
        "\n",
        "Commencez par cr\u00e9er un mod\u00e8le de r\u00e9f\u00e9rence qui servira de base de comparaison. Le mod\u00e8le `LinearClassifier` fournit un ensemble de *k* classificateurs un contre tous\u00a0; un pour chacune des *k* classes.\n",
        "\n",
        "Outre l'indication de la justesse et la repr\u00e9sentation graphique de la perte logistique au fil du temps, vous remarquerez qu'une [**matrice de confusion**](https://fr.wikipedia.org/wiki/Matrice_de_confusion) est \u00e9galement affich\u00e9e. Cette matrice identifie les classes qui ont \u00e9t\u00e9 class\u00e9es de mani\u00e8re erron\u00e9e. Quels chiffres ont \u00e9t\u00e9 confondus avec d'autres\u00a0?\n",
        "\n",
        "Notez \u00e9galement que la fonction `log_loss` est utilis\u00e9e pour le suivi de l'erreur du mod\u00e8le. Veillez \u00e0 ne pas la confondre avec la fonction de perte interne au mod\u00e8le `LinearClassifier`, laquelle est utilis\u00e9e \u00e0 des fins d'apprentissage."
      ]
    },
    {
      "metadata": {
        "id": "cpoVC4TSdw5Z",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def construct_feature_columns():\n",
        "  \"\"\"Construct the TensorFlow Feature Columns.\n",
        "\n",
        "  Returns:\n",
        "    A set of feature columns\n",
        "  \"\"\" \n",
        "  \n",
        "  # There are 784 pixels in each image. \n",
        "  return set([tf.feature_column.numeric_column('pixels', shape=784)])"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "kMmL89yGeTfz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " Vous allez maintenant cr\u00e9er des fonctions d'entr\u00e9e distinctes pour l'apprentissage et la pr\u00e9diction. Vous allez les imbriquer dans `create_training_input_fn()` et `create_predict_input_fn()`, respectivement, afin de pouvoir les invoquer pour renvoyer les fonctions `_input_fn` correspondantes qui doivent \u00eatre transmises aux appels `.train()` et `.predict()`."
      ]
    },
    {
      "metadata": {
        "id": "OeS47Bmn5Ms2",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def create_training_input_fn(features, labels, batch_size, num_epochs=None, shuffle=True):\n",
        "  \"\"\"A custom input_fn for sending MNIST data to the estimator for training.\n",
        "\n",
        "  Args:\n",
        "    features: The training features.\n",
        "    labels: The training labels.\n",
        "    batch_size: Batch size to use during training.\n",
        "\n",
        "  Returns:\n",
        "    A function that returns batches of training features and labels during\n",
        "    training.\n",
        "  \"\"\"\n",
        "  def _input_fn(num_epochs=None, shuffle=True):\n",
        "    # Input pipelines are reset with each call to .train(). To ensure model\n",
        "    # gets a good sampling of data, even when number of steps is small, we \n",
        "    # shuffle all the data before creating the Dataset object\n",
        "    idx = np.random.permutation(features.index)\n",
        "    raw_features = {\"pixels\":features.reindex(idx)}\n",
        "    raw_targets = np.array(labels[idx])\n",
        "   \n",
        "    ds = Dataset.from_tensor_slices((raw_features,raw_targets)) # warning: 2GB limit\n",
        "    ds = ds.batch(batch_size).repeat(num_epochs)\n",
        "    \n",
        "    if shuffle:\n",
        "      ds = ds.shuffle(10000)\n",
        "    \n",
        "    # Return the next batch of data.\n",
        "    feature_batch, label_batch = ds.make_one_shot_iterator().get_next()\n",
        "    return feature_batch, label_batch\n",
        "\n",
        "  return _input_fn"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "8zoGWAoohrwS",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def create_predict_input_fn(features, labels, batch_size):\n",
        "  \"\"\"A custom input_fn for sending mnist data to the estimator for predictions.\n",
        "\n",
        "  Args:\n",
        "    features: The features to base predictions on.\n",
        "    labels: The labels of the prediction examples.\n",
        "\n",
        "  Returns:\n",
        "    A function that returns features and labels for predictions.\n",
        "  \"\"\"\n",
        "  def _input_fn():\n",
        "    raw_features = {\"pixels\": features.values}\n",
        "    raw_targets = np.array(labels)\n",
        "    \n",
        "    ds = Dataset.from_tensor_slices((raw_features, raw_targets)) # warning: 2GB limit\n",
        "    ds = ds.batch(batch_size)\n",
        "    \n",
        "        \n",
        "    # Return the next batch of data.\n",
        "    feature_batch, label_batch = ds.make_one_shot_iterator().get_next()\n",
        "    return feature_batch, label_batch\n",
        "\n",
        "  return _input_fn"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "G6DjSLZMu8Um",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def train_linear_classification_model(\n",
        "    learning_rate,\n",
        "    steps,\n",
        "    batch_size,\n",
        "    training_examples,\n",
        "    training_targets,\n",
        "    validation_examples,\n",
        "    validation_targets):\n",
        "  \"\"\"Trains a linear classification model for the MNIST digits dataset.\n",
        "  \n",
        "  In addition to training, this function also prints training progress information,\n",
        "  a plot of the training and validation loss over time, and a confusion\n",
        "  matrix.\n",
        "  \n",
        "  Args:\n",
        "    learning_rate: A `float`, the learning rate to use.\n",
        "    steps: A non-zero `int`, the total number of training steps. A training step\n",
        "      consists of a forward and backward pass using a single batch.\n",
        "    batch_size: A non-zero `int`, the batch size.\n",
        "    training_examples: A `DataFrame` containing the training features.\n",
        "    training_targets: A `DataFrame` containing the training labels.\n",
        "    validation_examples: A `DataFrame` containing the validation features.\n",
        "    validation_targets: A `DataFrame` containing the validation labels.\n",
        "      \n",
        "  Returns:\n",
        "    The trained `LinearClassifier` object.\n",
        "  \"\"\"\n",
        "\n",
        "  periods = 10\n",
        "\n",
        "  steps_per_period = steps / periods  \n",
        "  # Create the input functions.\n",
        "  predict_training_input_fn = create_predict_input_fn(\n",
        "    training_examples, training_targets, batch_size)\n",
        "  predict_validation_input_fn = create_predict_input_fn(\n",
        "    validation_examples, validation_targets, batch_size)\n",
        "  training_input_fn = create_training_input_fn(\n",
        "    training_examples, training_targets, batch_size)\n",
        "  \n",
        "  # Create a LinearClassifier object.\n",
        "  my_optimizer = tf.train.AdagradOptimizer(learning_rate=learning_rate)\n",
        "  my_optimizer = tf.contrib.estimator.clip_gradients_by_norm(my_optimizer, 5.0)\n",
        "  classifier = tf.estimator.LinearClassifier(\n",
        "      feature_columns=construct_feature_columns(),\n",
        "      n_classes=10,\n",
        "      optimizer=my_optimizer,\n",
        "      config=tf.estimator.RunConfig(keep_checkpoint_max=1)\n",
        "  )\n",
        "\n",
        "  # Train the model, but do so inside a loop so that we can periodically assess\n",
        "  # loss metrics.\n",
        "  print(\"Training model...\")\n",
        "  print(\"LogLoss error (on validation data):\")\n",
        "  training_errors = []\n",
        "  validation_errors = []\n",
        "  for period in range (0, periods):\n",
        "    # Train the model, starting from the prior state.\n",
        "    classifier.train(\n",
        "        input_fn=training_input_fn,\n",
        "        steps=steps_per_period\n",
        "    )\n",
        "  \n",
        "    # Take a break and compute probabilities.\n",
        "    training_predictions = list(classifier.predict(input_fn=predict_training_input_fn))\n",
        "    training_probabilities = np.array([item['probabilities'] for item in training_predictions])\n",
        "    training_pred_class_id = np.array([item['class_ids'][0] for item in training_predictions])\n",
        "    training_pred_one_hot = tf.keras.utils.to_categorical(training_pred_class_id,10)\n",
        "        \n",
        "    validation_predictions = list(classifier.predict(input_fn=predict_validation_input_fn))\n",
        "    validation_probabilities = np.array([item['probabilities'] for item in validation_predictions])    \n",
        "    validation_pred_class_id = np.array([item['class_ids'][0] for item in validation_predictions])\n",
        "    validation_pred_one_hot = tf.keras.utils.to_categorical(validation_pred_class_id,10)    \n",
        "    \n",
        "    # Compute training and validation errors.\n",
        "    training_log_loss = metrics.log_loss(training_targets, training_pred_one_hot)\n",
        "    validation_log_loss = metrics.log_loss(validation_targets, validation_pred_one_hot)\n",
        "    # Occasionally print the current loss.\n",
        "    print(\"  period %02d : %0.2f\" % (period, validation_log_loss))\n",
        "    # Add the loss metrics from this period to our list.\n",
        "    training_errors.append(training_log_loss)\n",
        "    validation_errors.append(validation_log_loss)\n",
        "  print(\"Model training finished.\")\n",
        "  # Remove event files to save disk space.\n",
        "  _ = map(os.remove, glob.glob(os.path.join(classifier.model_dir, 'events.out.tfevents*')))\n",
        "  \n",
        "  # Calculate final predictions (not probabilities, as above).\n",
        "  final_predictions = classifier.predict(input_fn=predict_validation_input_fn)\n",
        "  final_predictions = np.array([item['class_ids'][0] for item in final_predictions])\n",
        "  \n",
        "  \n",
        "  accuracy = metrics.accuracy_score(validation_targets, final_predictions)\n",
        "  print(\"Final accuracy (on validation data): %0.2f\" % accuracy)\n",
        "\n",
        "  # Output a graph of loss metrics over periods.\n",
        "  plt.ylabel(\"LogLoss\")\n",
        "  plt.xlabel(\"Periods\")\n",
        "  plt.title(\"LogLoss vs. Periods\")\n",
        "  plt.plot(training_errors, label=\"training\")\n",
        "  plt.plot(validation_errors, label=\"validation\")\n",
        "  plt.legend()\n",
        "  plt.show()\n",
        "  \n",
        "  # Output a plot of the confusion matrix.\n",
        "  cm = metrics.confusion_matrix(validation_targets, final_predictions)\n",
        "  # Normalize the confusion matrix by row (i.e by the number of samples\n",
        "  # in each class).\n",
        "  cm_normalized = cm.astype(\"float\") / cm.sum(axis=1)[:, np.newaxis]\n",
        "  ax = sns.heatmap(cm_normalized, cmap=\"bone_r\")\n",
        "  ax.set_aspect(1)\n",
        "  plt.title(\"Confusion matrix\")\n",
        "  plt.ylabel(\"True label\")\n",
        "  plt.xlabel(\"Predicted label\")\n",
        "  plt.show()\n",
        "\n",
        "  return classifier"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ItHIUyv2u8Ur",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " **Exercez-vous, pendant cinq minutes, \u00e0 obtenir une valeur de justesse \u00e9lev\u00e9e avec un mod\u00e8le lin\u00e9aire de ce type. Pour cet exercice, limitez-vous \u00e0 exp\u00e9rimenter avec les hyperparam\u00e8tres relatifs \u00e0 la taille du lot, au taux d'apprentissage et aux nombres d'\u00e9tapes.**\n",
        "\n",
        "Arr\u00eatez-vous d\u00e8s que vous obtenez une valeur de justesse sup\u00e9rieure \u00e0 environ\u00a00,9."
      ]
    },
    {
      "metadata": {
        "id": "yaiIhIQqu8Uv",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "classifier = train_linear_classification_model(\n",
        "             learning_rate=0.02,\n",
        "             steps=100,\n",
        "             batch_size=10,\n",
        "             training_examples=training_examples,\n",
        "             training_targets=training_targets,\n",
        "             validation_examples=validation_examples,\n",
        "             validation_targets=validation_targets)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "266KQvZoMxMv",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ### Solution\n",
        "\n",
        "Cliquez ci-dessous pour afficher une solution."
      ]
    },
    {
      "metadata": {
        "id": "lRWcn24DM3qa",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " Voici un ensemble de param\u00e8tres avec lequel la justesse devrait avoisiner\u00a00,9."
      ]
    },
    {
      "metadata": {
        "id": "TGlBMrUoM1K_",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "_ = train_linear_classification_model(\n",
        "    learning_rate=0.03,\n",
        "    steps=1000,\n",
        "    batch_size=30,\n",
        "    training_examples=training_examples,\n",
        "    training_targets=training_targets,\n",
        "    validation_examples=validation_examples,\n",
        "    validation_targets=validation_targets)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "mk095OfpPdOx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ## T\u00e2che\u00a02\u00a0Remplacer le classificateur lin\u00e9aire par un r\u00e9seau de neurones\n",
        "\n",
        "**Remplacez le mod\u00e8le LinearClassifier ci-dessus par un mod\u00e8le [`DNNClassifier`](https://www.tensorflow.org/api_docs/python/tf/estimator/DNNClassifier) et identifiez une combinaison de param\u00e8tres offrant une justesse de 0,95 ou plus.**\n",
        "\n",
        "Vous pouvez essayer d'autres m\u00e9thodes de r\u00e9gularisation, telles qu'une r\u00e9gularisation par abandon. Ces m\u00e9thodes sont document\u00e9es dans les commentaires de la classe `DNNClassifier`."
      ]
    },
    {
      "metadata": {
        "id": "rm8P_Ttwu8U4",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "#\n",
        "# YOUR CODE HERE: Replace the linear classifier with a neural network.\n",
        "#"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "TOfmiSvqu8U9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " D\u00e8s que vous disposez d'un mod\u00e8le satisfaisant, v\u00e9rifiez que l'ensemble de validation n'a pas \u00e9t\u00e9 surappris en \u00e9valuant les donn\u00e9es de test qui seront charg\u00e9es ci-apr\u00e8s.\n"
      ]
    },
    {
      "metadata": {
        "id": "evlB5ubzu8VJ",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "mnist_test_dataframe = pd.read_csv(\n",
        "  \"https://download.mlcc.google.com/mledu-datasets/mnist_test.csv\",\n",
        "  sep=\",\",\n",
        "  header=None)\n",
        "\n",
        "test_targets, test_examples = parse_labels_and_features(mnist_test_dataframe)\n",
        "test_examples.describe()"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PDuLd2Hcu8VL",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "#\n",
        "# YOUR CODE HERE: Calculate accuracy on the test set.\n",
        "#"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "6sfw3LH0Oycm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ### Solution\n",
        "\n",
        "Cliquez ci-dessous pour afficher une solution."
      ]
    },
    {
      "metadata": {
        "id": "XatDGFKEO374",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " Le code ci-dessous est pratiquement identique au code d'apprentissage `LinearClassifer` d'origine, \u00e0 l'exception de la configuration sp\u00e9cifique au r\u00e9seau de neurones, comme l'hyperparam\u00e8tre des unit\u00e9s cach\u00e9es."
      ]
    },
    {
      "metadata": {
        "id": "kdNTx8jkPQUx",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "def train_nn_classification_model(\n",
        "    learning_rate,\n",
        "    steps,\n",
        "    batch_size,\n",
        "    hidden_units,\n",
        "    training_examples,\n",
        "    training_targets,\n",
        "    validation_examples,\n",
        "    validation_targets):\n",
        "  \"\"\"Trains a neural network classification model for the MNIST digits dataset.\n",
        "  \n",
        "  In addition to training, this function also prints training progress information,\n",
        "  a plot of the training and validation loss over time, as well as a confusion\n",
        "  matrix.\n",
        "  \n",
        "  Args:\n",
        "    learning_rate: A `float`, the learning rate to use.\n",
        "    steps: A non-zero `int`, the total number of training steps. A training step\n",
        "      consists of a forward and backward pass using a single batch.\n",
        "    batch_size: A non-zero `int`, the batch size.\n",
        "    hidden_units: A `list` of int values, specifying the number of neurons in each layer.\n",
        "    training_examples: A `DataFrame` containing the training features.\n",
        "    training_targets: A `DataFrame` containing the training labels.\n",
        "    validation_examples: A `DataFrame` containing the validation features.\n",
        "    validation_targets: A `DataFrame` containing the validation labels.\n",
        "      \n",
        "  Returns:\n",
        "    The trained `DNNClassifier` object.\n",
        "  \"\"\"\n",
        "\n",
        "  periods = 10\n",
        "  # Caution: input pipelines are reset with each call to train. \n",
        "  # If the number of steps is small, your model may never see most of the data.  \n",
        "  # So with multiple `.train` calls like this you may want to control the length \n",
        "  # of training with num_epochs passed to the input_fn. Or, you can do a really-big shuffle, \n",
        "  # or since it's in-memory data, shuffle all the data in the `input_fn`.\n",
        "  steps_per_period = steps / periods  \n",
        "  # Create the input functions.\n",
        "  predict_training_input_fn = create_predict_input_fn(\n",
        "    training_examples, training_targets, batch_size)\n",
        "  predict_validation_input_fn = create_predict_input_fn(\n",
        "    validation_examples, validation_targets, batch_size)\n",
        "  training_input_fn = create_training_input_fn(\n",
        "    training_examples, training_targets, batch_size)\n",
        "  \n",
        "  # Create the input functions.\n",
        "  predict_training_input_fn = create_predict_input_fn(\n",
        "    training_examples, training_targets, batch_size)\n",
        "  predict_validation_input_fn = create_predict_input_fn(\n",
        "    validation_examples, validation_targets, batch_size)\n",
        "  training_input_fn = create_training_input_fn(\n",
        "    training_examples, training_targets, batch_size)\n",
        "  \n",
        "  # Create feature columns.\n",
        "  feature_columns = [tf.feature_column.numeric_column('pixels', shape=784)]\n",
        "\n",
        "  # Create a DNNClassifier object.\n",
        "  my_optimizer = tf.train.AdagradOptimizer(learning_rate=learning_rate)\n",
        "  my_optimizer = tf.contrib.estimator.clip_gradients_by_norm(my_optimizer, 5.0)\n",
        "  classifier = tf.estimator.DNNClassifier(\n",
        "      feature_columns=feature_columns,\n",
        "      n_classes=10,\n",
        "      hidden_units=hidden_units,\n",
        "      optimizer=my_optimizer,\n",
        "      config=tf.contrib.learn.RunConfig(keep_checkpoint_max=1)\n",
        "  )\n",
        "\n",
        "  # Train the model, but do so inside a loop so that we can periodically assess\n",
        "  # loss metrics.\n",
        "  print(\"Training model...\")\n",
        "  print(\"LogLoss error (on validation data):\")\n",
        "  training_errors = []\n",
        "  validation_errors = []\n",
        "  for period in range (0, periods):\n",
        "    # Train the model, starting from the prior state.\n",
        "    classifier.train(\n",
        "        input_fn=training_input_fn,\n",
        "        steps=steps_per_period\n",
        "    )\n",
        "  \n",
        "    # Take a break and compute probabilities.\n",
        "    training_predictions = list(classifier.predict(input_fn=predict_training_input_fn))\n",
        "    training_probabilities = np.array([item['probabilities'] for item in training_predictions])\n",
        "    training_pred_class_id = np.array([item['class_ids'][0] for item in training_predictions])\n",
        "    training_pred_one_hot = tf.keras.utils.to_categorical(training_pred_class_id,10)\n",
        "        \n",
        "    validation_predictions = list(classifier.predict(input_fn=predict_validation_input_fn))\n",
        "    validation_probabilities = np.array([item['probabilities'] for item in validation_predictions])    \n",
        "    validation_pred_class_id = np.array([item['class_ids'][0] for item in validation_predictions])\n",
        "    validation_pred_one_hot = tf.keras.utils.to_categorical(validation_pred_class_id,10)    \n",
        "    \n",
        "    # Compute training and validation errors.\n",
        "    training_log_loss = metrics.log_loss(training_targets, training_pred_one_hot)\n",
        "    validation_log_loss = metrics.log_loss(validation_targets, validation_pred_one_hot)\n",
        "    # Occasionally print the current loss.\n",
        "    print(\"  period %02d : %0.2f\" % (period, validation_log_loss))\n",
        "    # Add the loss metrics from this period to our list.\n",
        "    training_errors.append(training_log_loss)\n",
        "    validation_errors.append(validation_log_loss)\n",
        "  print(\"Model training finished.\")\n",
        "  # Remove event files to save disk space.\n",
        "  _ = map(os.remove, glob.glob(os.path.join(classifier.model_dir, 'events.out.tfevents*')))\n",
        "  \n",
        "  # Calculate final predictions (not probabilities, as above).\n",
        "  final_predictions = classifier.predict(input_fn=predict_validation_input_fn)\n",
        "  final_predictions = np.array([item['class_ids'][0] for item in final_predictions])\n",
        "  \n",
        "  \n",
        "  accuracy = metrics.accuracy_score(validation_targets, final_predictions)\n",
        "  print(\"Final accuracy (on validation data): %0.2f\" % accuracy)\n",
        "\n",
        "  # Output a graph of loss metrics over periods.\n",
        "  plt.ylabel(\"LogLoss\")\n",
        "  plt.xlabel(\"Periods\")\n",
        "  plt.title(\"LogLoss vs. Periods\")\n",
        "  plt.plot(training_errors, label=\"training\")\n",
        "  plt.plot(validation_errors, label=\"validation\")\n",
        "  plt.legend()\n",
        "  plt.show()\n",
        "  \n",
        "  # Output a plot of the confusion matrix.\n",
        "  cm = metrics.confusion_matrix(validation_targets, final_predictions)\n",
        "  # Normalize the confusion matrix by row (i.e by the number of samples\n",
        "  # in each class).\n",
        "  cm_normalized = cm.astype(\"float\") / cm.sum(axis=1)[:, np.newaxis]\n",
        "  ax = sns.heatmap(cm_normalized, cmap=\"bone_r\")\n",
        "  ax.set_aspect(1)\n",
        "  plt.title(\"Confusion matrix\")\n",
        "  plt.ylabel(\"True label\")\n",
        "  plt.xlabel(\"Predicted label\")\n",
        "  plt.show()\n",
        "\n",
        "  return classifier"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ZfzsTYGPPU8I",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "classifier = train_nn_classification_model(\n",
        "    learning_rate=0.05,\n",
        "    steps=1000,\n",
        "    batch_size=30,\n",
        "    hidden_units=[100, 100],\n",
        "    training_examples=training_examples,\n",
        "    training_targets=training_targets,\n",
        "    validation_examples=validation_examples,\n",
        "    validation_targets=validation_targets)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "qXvrOgtUR-zD",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " Vous allez ensuite v\u00e9rifier la justesse de l'ensemble d'\u00e9valuation."
      ]
    },
    {
      "metadata": {
        "id": "scQNpDePSFjt",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "mnist_test_dataframe = pd.read_csv(\n",
        "  \"https://download.mlcc.google.com/mledu-datasets/mnist_test.csv\",\n",
        "  sep=\",\",\n",
        "  header=None)\n",
        "\n",
        "test_targets, test_examples = parse_labels_and_features(mnist_test_dataframe)\n",
        "test_examples.describe()"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "EVaWpWKvSHmu",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "predict_test_input_fn = create_predict_input_fn(\n",
        "    test_examples, test_targets, batch_size=100)\n",
        "\n",
        "test_predictions = classifier.predict(input_fn=predict_test_input_fn)\n",
        "test_predictions = np.array([item['class_ids'][0] for item in test_predictions])\n",
        "  \n",
        "accuracy = metrics.accuracy_score(test_targets, test_predictions)\n",
        "print(\"Accuracy on test data: %0.2f\" % accuracy)"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "WX2mQBAEcisO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " ## T\u00e2che\u00a03\u00a0: Visualiser les pond\u00e9rations de la premi\u00e8re couche cach\u00e9e\n",
        "\n",
        "Passez quelques minutes \u00e0 explorer le r\u00e9seau de neurones pour savoir ce qu'il a appris. Pour cela, vous allez acc\u00e9der \u00e0 l'attribut `weights_` du mod\u00e8le.\n",
        "\n",
        "La couche d'entr\u00e9e du mod\u00e8le compte `784`\u00a0pond\u00e9rations correspondant aux images d'entr\u00e9e `28\u00d728`. La premi\u00e8re couche cach\u00e9e aura `784\u00d7N`\u00a0pond\u00e9rations, o\u00f9 `N` repr\u00e9sente le nombre de n\u0153uds de cette couche. Vous pouvez reconvertir ces pond\u00e9rations en images `28\u00d728` en *remodelant* chacun des `N` tableaux de pond\u00e9rations `1\u00d7784` en `N`\u00a0tableaux d'une taille de `28\u00d728`.\n",
        "\n",
        "Ex\u00e9cutez la cellule suivante pour repr\u00e9senter graphiquement les pond\u00e9rations. Pour cette cellule, notez qu'un mod\u00e8le `DNNClassifier` appel\u00e9 \"classifier\" doit d\u00e9j\u00e0 avoir \u00e9t\u00e9 entra\u00een\u00e9."
      ]
    },
    {
      "metadata": {
        "id": "eUC0Z8nbafgG",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "test": {
            "output": "ignore",
            "timeout": 600
          }
        },
        "cellView": "both"
      },
      "source": [
        "print(classifier.get_variable_names())\n",
        "\n",
        "weights0 = classifier.get_variable_value(\"dnn/hiddenlayer_0/kernel\")\n",
        "\n",
        "print(\"weights0 shape:\", weights0.shape)\n",
        "\n",
        "num_nodes = weights0.shape[1]\n",
        "num_rows = int(math.ceil(num_nodes / 10.0))\n",
        "fig, axes = plt.subplots(num_rows, 10, figsize=(20, 2 * num_rows))\n",
        "for coef, ax in zip(weights0.T, axes.ravel()):\n",
        "    # Weights in coef is reshaped from 1x784 to 28x28.\n",
        "    ax.matshow(coef.reshape(28, 28), cmap=plt.cm.pink)\n",
        "    ax.set_xticks(())\n",
        "    ax.set_yticks(())\n",
        "\n",
        "plt.show()"
      ],
      "cell_type": "code",
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "kL8MEhNgrx9N",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " La premi\u00e8re couche cach\u00e9e du r\u00e9seau de neurones doit normalement mod\u00e9liser des caract\u00e9ristiques de bas niveau. Lorsque vous visualiserez les pond\u00e9rations, vous ne verrez donc probablement que quelques blobs flous ou peut-\u00eatre quelques parties de chiffres. Vous constaterez \u00e9galement que certains neurones sont essentiellement des donn\u00e9es bruit\u00e9es\u00a0; soit ils n'ont pas converg\u00e9, soit ils sont ignor\u00e9s par les couches sup\u00e9rieures.\n",
        "\n",
        "Il peut \u00eatre int\u00e9ressant d'interrompre l'apprentissage apr\u00e8s un certain nombre d'it\u00e9rations afin d'examiner le r\u00e9sultat.\n",
        "\n",
        "**Effectuez l'apprentissage du classificateur pour 10, 100 et 1\u00a0000\u00a0pas, puis ex\u00e9cutez \u00e0 nouveau cette visualisation.**\n",
        "\n",
        "Quelles diff\u00e9rences remarquez-vous pour les diff\u00e9rents niveaux de convergence\u00a0?"
      ]
    }
  ]
}
